Workaround for large object size #755

ycl6 · 2024-12-12T17:55:26Z

This PR is to solve the problem when dealing with large object size, by casting total_size as a numeric object when it is stored as an attribute in globals.

When the integer class total_size exceed the .Machine$integer.max limit, users will receive the integer overflow warning message when FutureGlobals calculates size as I quoted below, and size will be NA as the result.

Warning message in size + sum(size_args, na.rm = FALSE):
“NAs produced by integer overflow”

future/R/FutureGlobals-class.R

Lines 85 to 102 in 568e39a

    
             ## Total size? 
        
             size <- attr(x, "total_size", exact = TRUE) 
        
             if (!is.na(size)) { 
        
               size_args <- lapply(args, FUN = function(z) { 
        
                 size <- attr(z, "total_size", exact = TRUE) 
        
                 if (is.null(size)) NA_real_ else size 
        
               }) 
        
               size_args <- unlist(size_args, use.names = FALSE) 
        
               size <- size + sum(size_args, na.rm = FALSE) 
        
             } 
        
             x <- NextMethod() 
        
             attr(x, "resolved") <- resolved 
        
             attr(x, "total_size") <- size 
        
             x 
        
           }

This has been reported when using other R packages that use future for parallelisation, for example satijalab/seurat#9316, satijalab/seurat#9484, jinworks/CellChat#250.

HenrikBengtsson · 2024-12-15T18:00:27Z

Thanks for the report and for your troubleshooting.

This problem was introduced in future 1.34.0 (2024-07-29), when we switched from utils::object.size() to parallelly::serializedSize() to calculate object sizes.

Although we could work around it here in future, I think a more reliable approach is to fix parallelly::serializedSize() as planned in futureverse/parallelly#123.

PS. I think an integer overflow can happen even before the location where you suggested the fix, namely in:

future/R/globals.R

Line 359 in a5a3248

total_size <- sum(sizes, na.rm = TRUE)

Closing this PR because it will be fixed in the parallelly package instead.

ycl6 · 2024-12-15T18:41:57Z

Hi @HenrikBengtsson Thanks for nailing the source for making the changes!

Workaround for large object size

11d04a5

HenrikBengtsson mentioned this pull request Dec 15, 2024

serializedSize(): Always return a double (to lower the risk for integer overflow downstream) futureverse/parallelly#123

Closed

HenrikBengtsson closed this Dec 15, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Workaround for large object size #755

Workaround for large object size #755

ycl6 commented Dec 12, 2024

HenrikBengtsson commented Dec 15, 2024 •

edited

Loading

ycl6 commented Dec 15, 2024

	## Total size?
	size <- attr(x, "total_size", exact = TRUE)
	if (!is.na(size)) {
	size_args <- lapply(args, FUN = function(z) {
	size <- attr(z, "total_size", exact = TRUE)
	if (is.null(size)) NA_real_ else size
	})
	size_args <- unlist(size_args, use.names = FALSE)
	size <- size + sum(size_args, na.rm = FALSE)
	}

	x <- NextMethod()

	attr(x, "resolved") <- resolved
	attr(x, "total_size") <- size

	x
	}

Workaround for large object size #755

Workaround for large object size #755

Conversation

ycl6 commented Dec 12, 2024

HenrikBengtsson commented Dec 15, 2024 • edited Loading

ycl6 commented Dec 15, 2024

HenrikBengtsson commented Dec 15, 2024 •

edited

Loading